A Kernel-Based Communication Fault Injector for Dependability Testing of Distributed Systems

Software-implemented fault injection is a powerful strategy to test fault-tolerant protocols in distributed environments. In this paper, we present ComFIRM, a communication fault injection tool we developed which minimizes the probe effect on the tested protocols. ComFIRM explores the possibility to insert code directly inside the Linux kernel in the lowest level of the protocol stack through the load of modules. The tool injects faults directly into the message exchange subsystem, allowing the definition of test scenarios from a wide fault model that can affect messages being sent and/or received. Additionally, the tool is demonstrated in an experiment which applies the fault injector to evaluate the behavior of a group membership service under communication faults.